Multimodal Discourse: In Search of Units

Human communication is inherently multimodal. In this study we focus on three channels of spoken discourse: the verbal component, prosody, and gesticulation. We address the question of units that can be identified within these components and in spoken multimodal discourse as a whole. The basic unit of the verbal channel is the clause, reporting an event or a state. A set of prosodic criteria help to define elementary discourse units, that is prosodic units serving as quanta of discourse production. The gestural channel consists of individual gestures, each defined by a set of features. Elementary discourse units are strongly coordinated with both clauses and gestures and can thus be considered basic units of multimodal discourse. Larger units can also be identified, such as prosodic sentences and series of gestures that again demonstrate coordination. By identifying units of natural discourse, coordinated across various channels, we make a step towards multimodal linguistics.